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(54) Method of incorporating a variable rate auxiliary data stream with a variable rate primary 
data stream 

(57) A method of incorporating an auxiliary signal with an input video signal includes the step of encoding the input 
video signal to generate a variable bit rate encoded signal having a selected level of distortion. The video signal is 
applied to a buffer The level of distortion is selected as a function of the fullness of the buffer. The video signal is read 
from the buffer and applied to a transmission channel at a predetermined video output rate. Portions of an auxiliary 
signal are applied to the channel at times when the buffer is in an underflow condition. The selected level of distortion 
has a flow which corresponds to a predetermined, non-zero level of perceptual distortion in the video signal. 
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Description 

Background of the Invention 

The present invention relates to the transmission of s 
primary and auxiliary digital data streams over a broad- 
cast channel- 
Systems, such as HDTV systems, use encoding 
techniques to extract redundancies from video signals 
which are transmitted as video frames from a transmitter 
to a receiver The amount of information contained in the 
encoded video signals tends to be variable, resulting in 
a bursty signal. For example, scenes involving a lot of 
action, such as basketball footage, contain less 
frame-to-frame redundant information, and therefore 
tend to produce a larger amount of encoded information 
or data, than the amount of encoded data required to 
transmit subsequent frames of a still picture, which con- 
tain primarily redundant information. 

For transmission using a limited-brt-rate channel, an 
HDTV encoder must produce a constrained constant bit 
rate. In order to ensure a constant bit rate output, a buffer 
is associated with the HDTV transmitter which accumu- 
lates the bursty video signals and controls the rate at 
which the bitstream is transmitted. If there is a break in 
the transmission of the bitstream generated by the HDTV 
transmitter such that the buffer becomes empty, partial 
or total loss of picture results. 

In an HDTV system, video data are received from a 
data source and compressed by a vrcteo encoder Among 
other things, the video encoder determines the level, i.e. 
coarseness of quantization of the video data stream. The 
video data are then transmitted to a buffer The buffer 
controls the output rate of the video data stream. The 
fullness of the buffer is controlled by distortion control via 
a feedback mechanism. The distortion control controls 
the level of distortion, i.e., the coarseness or fineness of 
the video data, for each video frame which, in turn, ef- 
fects the level of, i.e.. makes more coarse, the quantiza- 
tion of the encoded video data. When the buffer fills up 
toa pointatx)ve some reference level, the distortion con- 
trol causes the ^coder to increase the level of quanti- 
zation of the video data stream, thereby reducing buffer 
fullness. When the buffer fullness falls below some ref- 
erence level, the distortion control causes the encoder 
to reduce the level of quantizatbn of the vkieo data 
stream, thereby increasing buffer fullness. In the event 
of buffer underflow, i.e. , so many more bits leave the buff- 
er than enter the buffer that the buffer empties. 

Summary of the Invention 

In the prior art, the buffer as a practical matter rarely 
empties even when a highly redundant image, such as 
a still image, is being sent. The reason is that even a still 
image contains noise which gives rise to some 
frame-to-frame differences which cause the encoder to 
increase the level of quantization. 



We have recognized that by establishing a fkx)r on 
the allowed level dt distortion, the buffer tends to more 
frequently realize an underflow conditkxi. Auxiliary data 
representative of data services, such as electronic pub- 
lk:ations, can be transmitted during the underftow condi- 
tion. While having a floor on the distortbn level tends to 
increase the level of quantization of the video data 
stream compared to what a prk>r art system would do 
resulting in reduced picture quality over that which the 
prior art could achieve, the floor Is established at such a 
level that the picture quality is acceptable to the human 
eye. 

Brief Description of the Drawing 

FIG. 1 illustrates a schematic diagram of an HDTV 
system implemented in accordance with the present in- 
vention. 

FIG. 2 illustrates a block diagram of the HDTV trans- 
mitter of the HDTV system of FIG. 1 . 

FIG. 3 illustrates a block diagram of the HDTV re- 
ceiver of the HDTV system of FIG. 1 . 

FIG. 4 illustrates a graph depicting a measurement 
of the distortk>n level over time t. 

FIG. 5 illustrates a graph depicting the relationship 
between the level of distortion and bit rate. 

Detailed Description 

A general overview of a video system, illustratively 
a high definition television (HDTV) system 10, incorpo- 
rating the principles of the present invention is illustrated 
in FIG. 1 . The details of an HDTV vkJeo system are de- 
scribed in such U.S. Patents as 5.144,423 entitled 
■HDTV Encoder with Fonward Estimation and Constant 
Flate Motbn Vectors" issued September 1, 1992 to 
Knauer et al.; 5,136,377 entitled "Adaptive Non-linear 
Quantizer", issued August 4, 1992 to Johnston et al.; 
5,134,475 entitled "Adaptive Leak HDTV Encoder", is- 
sued July 28, 1 992 to Johnston et al.; and 5, 1 34,477 en- 
titled "HDTV Receiver" issued July 28, 1992 to Knauer 
et al. 

In operation, a viewer sends instructions to a central 
data server 12 to identify one or more data servk;es that 
are desired to be received. The instructbns are transmit- 
ted from the viewer's remote locatran to the central serv- 
er 1 2 by means of a cable or fiber optic network 1 4. For 
example, the instructions might be transmitted as in- 
fra-red signals via a remote control unit 16 to a set4op 
box 18 associated with receiver 30 which transmits the 
appropriate signals over the network 14. Alternatively, 
the viewer could transmit his instructions to central serv- 
er 1 2 over a standard telephone network (not shown). A 
separate data service converter box (not shown) may be 
used in place of the set-top box 1 8 to transmit the instruc- 
tions over network 14. 

The central sender 12 processes the incoming re- 
quests from the viewers and retrieves the data desired 
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by the viewers from a respective one of servers 24. TTie 
data services can be any type of video or audio informa- 
tion such as, but not limited to, electronic newspapers, 
magazines, catalogs, movies transmitted to a storage 
device, or other types of services which are not required 
to be transmitted in real time. The data is transmitted to 
the set-top box 18. The set-top box 18 includes storage 
means, such as menrK)ry, for storing the data. Once a 
significant portion of the data has been transmitted to the 
set-top box 18, the service can be viewed by the viewer. 
Set-top box also receives video signals over network 14 
from a central HDTV transmitter 20. 

While the data is being transmitted to the viewer's 
set-top box 18, the viewer is still capable of viewing pro- 
grams broadcast from the HDTV transmitter 20. In ac- 
cordance with the present invention, HDTV video signals 
and the data signals are multiplexed together in such a 
manner that the HDTV video signals are transmitted in 
real time and the data service signals are transmitted to 
the set-top box at a sbwer rate. As such, the HDTV sig- 
nals are the primary signals broadcast to the set-top box 
18 arKJ the data signals are treated as auxiliary signals 
as will be described in detail hereinafter. 

FIG. 2 is a more detailed block diagram of the HDTV 
transmitter 20 shown in FIG. 1. A high-definition (HD) vid- 
eo signal on lead 205 is encoded by an HDTV encoder 
210. As is well known, an HDTV encoder-indeed, most 
video encoders-provides a variable-bit-rate output de- 
pending on the nature of the video signal being encoded. 
For example, a video signal which is changing by a rel- 
atively small amount from frame to frame can be encod- 
ed with fewer bits than when the frame to frame differ- 
ences are relatively great, as when there is a lot of motion 
in the image. A sharp scene change also causes the en- 
coder output bit rate to increase for a time. However, 
while small scene changes require fewer bits than drastic 
scene changes, noise present in the video signals caus- 
es the image to become distorted. As a result, the 
number of bits required to encode the scene changes is 
significant. 

As a result of the above-described variability in the 
output bit rate of encoder 21 0, its output is buffered in a 
video data buffer 215 so that an appropriately fonmatted 
HDTV signal can be developed and applied to the broad- 
cast channel 220. Interposed between video data buffer 
215 and channel 220 is a multiplexer 225 whose function 
will be explained nnomentarily. 

The system also includes distortion control 230 
which receives as its inputs the incoming video signal 
from lead 205 as well as a 'buffer fullness* signal on lead 
235 provided by buffer 215. The function and structure 
of distortion control 230 are well known in the art. The 
distortion control 230 determines the degree of coarse- 
ness to which a video signal is encoded-its distortion lev- 
el-taking into consideration factors, such as, but not lim- 
ited to, discrete cosine transform (DCT) coefficients, mo- 
tion vectors and the level of quantization. The coded vid- 
eo signal must contain enough informatbn so that the 
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coded innage maintains the integrity of the original image. 
However, that measurement of integrity is subject to the 
constraint that the coded image will be viewed by a hu- 
man observer who will not be aware of snr^all distortions 
5 in the coded innage. As kxig as the human observer can 
not detect distortk)ns resulting from the coding of the im- 
age, the distortion level is considered to be within an ac- 
ceptable range. 

The distortion level is dependent on a perceptual 
10 distort bn metric. The perceptual distortion metric pro- 
vkies a measurement by which the degree of fineness 
to which the video signals are encoded nnay be deter- 
mined so that the distortk>n present in the coded image 
appears, to a human obsewer, to be uniformly distribut- 
es ed. In accordance with the present inventk)n, the percep- 
tual distortion metric is set at such a level that a floor is 
established for the distortbn level. Thus, while the image 
may be capable of being more finely encoded, such en- 
coding wouki not be perceptible to the human observer. 
20 The relationship between the distortion level and the 
bit rate of the encoded video signal is illustrated in FIG. 
5, The distortion level and the bit rate have a generally 
inverse relationship. When the distortion level is high, the 
rate is bw. When the distortion level is low. the rate is 
25 high. By setting a perceptual distortion metric whteh lim- 
its how "undistorted" the video is made to be, thereby 
establishing a distortbn floor, the maximum amount of 
encoding, i.e., the degree of fineness to whk;h the video 
signal is encoded, can be limited. As shown in FIG. 4. 
30 over time the distortion level tends to decrease. Howev- 
er, when the distortbn floor is reached, the degree of 
fineness of the encoding associated with the distortbn 
level is maintained until the video image changes or 
there is an increase in scene complexity, at which point 
35 the distortion level is allowed to increase in order to ac- 
commodate the resulting increase in the coded bit rate. 

A consequence of establishing a floor on the distor- 
tion level is that over time, the buffer fullness will de- 
crease. In accordance with the present invention, the dis- 
40 tortion floor, which corresponds to predetermined, 
non-zero level of perceptual distortion in the video signal, 
is set at such a level that the buffer will regularty achieve 
an underflow condition. During the underflow condition, 
auxiliary data from auxiliary data buffer 245 are transmit- 
45 ted and transmission of the video signals is suspended. 
When the buffer fills above a predetermined level, trans- 
missbn of auxiliary data is suspended and transmission 
of the video signals is resumed. 

By creating artificial underflow conditions in this way, 
50 the anrK>unt of auxiliary data whbh may be transmitted is 
increased over the prior art. In partbular, at various 
points in the encoding of the vkJeo signal, such as during 
"stills' and periods of relatively slow movement, and/or 
when particular types of vbeo signals are being encod- 
es ed, such as 24- or 30-f rame/second film, the full data rate 
capability of the channel nnay not be taken up by the 
number of bits being supplied to, and ultimately read out 
of, video data buffer 21 5. The system thus has the ability 
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during such periods to accommodate the transmission 
ot auxiliary data that would not otherwise be able to be 
accommodated. 

Specifically, then, the system receives an auxiliary 
data bit stream on lead 240. The bits of this data stream 
are buttered in auxiliary data buffer 245 whose output 
feeds into multiplexer 225. In accordance with one em- 
bodiment, the system may provide a minimum constant 
bit-rate capability to establish a minimum brt-rate of aux- 
iliary data by dedicating a portion of the video channel 
exclusively to the transmission of auxiliary data. That is, 
bits from auxiliary data buffer 245 are continuously read 
therefrom at at least a predetermined minimum rate and 
multiplexed with the HDTV-encoded signal for presenta- 
tion to channel 220. However, as described above, when 
the video data buffer is empty~or, perhaps in other em- 
t>odiments, almost empty-indicating that the channel ca- 
pacity is not being fully utilized by the current combined 
level of bits being output by HDTV encoder 210 and aux- 
iliary data buffer 245, multiplexer 225 increases the rate 
at which it takes auxiliary data bits from buffer 245. Later, 
as video data buffer 21 5 begins to filt, the rate of auxiliary 
data taken from data buffer 245 is decreased-uttimately 
to the aforementioned minimum constant bit rate. 

FIG. 3 further illustrates a more detailed block dia- 
gram of the HDTV receiver 30. At the far end of the chan- 
nel, the composite video/auxiliary data signal is demul- 
tiplexed by demultiplexer 305. The received HDTV video 
signal is applied to video data buffer 31 0, then to HDTV 
decoder 315, and ultimately to a display 320 while the 
received auxiliary data is applied to auxiliary data buffer 
325 and then to a storage device 330. That storage de- 
vice 330 might be the display 320 on which the HDTV 
signal is displayed or could be a virtual set-top box which 
can transmit the data to any kind of end-use device. 

The broadcast industry has indicated that digital 
data services will be a valuable part of HDTV. Advanta- 
geously, this arrangement provkles a useful way of de- 
livering such services. In particular, the ability to transmit 
auxiliary data abng with the HDTV signal provides the 
capability of delivering such services as electronic news- 
papers and magazines, electronic catak>gues, paging 
services, stock quotes and program guides, as well as 
additkxial audb and/or video channels. 

It will be appreciated that those skilled in the art will 
be able to devise numerous and various alternative ar- 
rangements which, although not explicitly shown or de- 
scribed herein, embody the principles of the invention 
and are within its scope. 

Claims 

1 . A method comprising the steps of: 

encoding an input video signal to generate a 
variable-bit-rate encoded signal having a selected 
level of distort bn. 

applying said video signal to a buffer, said level 



of distortbn being selected as a f unctk>n of the full- 
ness of said buffer, 

reading said video signal from said buffer and 
applying it to a transmissbn channel at a predeter- 
5 mined video output rate, and 

applying portbns of an auxiliary signal to said 
channel at times when said buffer is in an underflow 
conditbn, 

said selected level of distortbn having a floor 
10 which corresponds to a predetermined, non-zero 
level of perceptual distortbn in said video signal, 

2. The method of claim 1 wherein said predetermined, 
non-zero level of perceptual distortbn is such that 

IS underftow condition occurs often enough that said 
auxiliary signal can be applied to said channel at at 
least a predetermined auxiliary data rate. 

3. The method of claim 1 wherein said step of encoding 
20 an input video signal comprises the step of: 

generating a perceptual distortion metric indic- 
ative of a desired level of distortion for the encoded 
signal, said perceptual distortion metric being a 
functbn of the fullness of said buffer. 

25 

4. A method comprising the steps of: 

encoding an input video signal to generate a 
variable-bit-rate encoded signal, said encoding 
including the steps of generating a perceptual dis- 

50 tortbn metric indicative of a desired level of distor- 
tion for the encoded signal and encoding said input 
video signal in such a way as to achieve sab desired 
level of distortion. 

applying said video signal to a buffer, said per- 

35 ceptual distortbn metric being a function of the full- 
ness of said buffer, 

reading said video signal from said buffer and 
applying it to a transmissbn channel at a predeter- 
mined video output rate, and 

40 applying portions of an auxiliary signal to said 

channel at times when said buffer is in an underflow 
conditbn, 

wherein sab perceptual distortion metric has 
a floor whose value is such that the application of 
45 said auxiliary signal to sab channel occurs at a pre- 
determined average bit rate. 

5. The method of claim 4 wherein sab floor is a function 
of the level of distortion of said video signal at which 

50 a lower level dt distortion will not be perceived by a 
human viewer. 

6. The method of claim 4 wherein said floor is substan- 
tially at a level of distortion is not perceived by a 

ss human viewer. 

7. A method of incorporating an auxiliary signal with a 
variable rate primary signal comprising the steps c^: 
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encoding the prinr^ry signal at a particular 
level of quantization; applying said primary encoded 
signal to a buffer; 

transmitting said primary signal over a trans- 
mission channel at a predetermined output rate; 5 

setting the level of quantization so that an 
underflow condition is achieved in said buffer; and 

transmitting the variable auxiliary data stream 
over said transmission channel during said under- 
flow condition. io 

8. The method according to claim 7 wherein said step 
of setting the level of quantization comprises the 
steps of: 

establishing a perceptual distortion metric 
which Is responsive to the state of buffer fullness; 

measuring a distortion level associated with 
the primary signal; and 

establishing a floor for the distortion level by 
lowering the perceptual distortion metric, thereby 
causing the buffer to achieve an underflow condi- 
tion. 

9. The method according to claim 8 wherein said step 

of establishing a floor comprises adjusting the per- ^5 
ceptual distortion metric so that the buffer empties 
more frequently resulting in more frequent buffer 
underflow conditions and rru^re frequent transmis- 
sion of the auxiliary signals. 
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